Identification of distributed user interface (dui) elements

ABSTRACT

Technologies are generally described to develop and implement a searchable knowledge source to identify distributed user interface (DUI) elements. In some examples, a DUI identification system may receive a control record of an application and populate one or more searchable knowledge sources based on an application description retrieved. The application description may include keywords, input elements, and output elements, and the searchable knowledge sources may be generated from control records of a multitude of applications. The DUI identification system may execute a query on the searchable knowledge sources based on the received keywords, input elements, and output elements associated with a target workflow from a requesting client. A query result that includes one or more DUI elements may be provided to the requesting client. The DUI elements may connect the input elements to corresponding output elements and match the keywords associated with the target workflow.

BACKGROUND

Unless otherwise indicated herein, the materials described in thissection are not prior art to the claims in this application and are notadmitted to be prior art by inclusion in this section.

Web pages and applications are largely identified through implementationof web search engines and application search engines, respectively.However, techniques to identify distributed user interface (DUI)elements have not received much attention, specifically how a client mayidentify DUI elements with which they may want to populate theirpersonal devices, widget panels, web pages, and the like.

A prevalent feature of a DUI system is the ability to link togethermultiple DUI elements so that a control panel and/or a collection canperform actions that a single application may not be able accomplishalone, such as by remotely operating multiple applications. For example,a user may set up an image processing macro to paste and modify apicture in an image processing program and then export a button via aDUI add-on, an operating system (OS) function, or an overlay framework.A user may then group the button with another button that captures animage (for example, from a hardware camera or a web page) and copy theother button to the clipboard. Using both buttons may enable the user toeffectively capture an image from a first source using a first interfaceelement, deliver the image as a source to a second interface element,and transpire with the output from the second interface element.However, without pre-loading or similar identification of various DUIelements, the user may not be able to find, implement, and linkdifferent DUI elements suitable for particular scenarios based on deviceconfigurations, network connections, application types, etc.

SUMMARY

The present disclosure generally describes techniques to identifydistributed user interface (DUI) elements through development andimplementation of a searchable knowledge source.

According to some examples, a method to develop a searchable knowledgesource for identification of distributed user interface (DUI) elementsis provided. An example method may include receiving a control record ofa source application and retrieving an application descriptionassociated with the source application. The method may further includepopulating one or more searchable knowledge sources based on theapplication description.

According to other examples, a method to identify distributed userinterface (DUI) elements through a searchable knowledge source isprovided. An example method may include receiving keywords, inputelements, and output elements associated with a target workflow. Themethod may also include executing a query on the searchable knowledgesource with query terms based on the received keywords, input elements,and output elements, where the searchable knowledge source may begenerated from control records of a plurality of applications. Themethod may further include providing a query result to a requestingclient, the query result including one or more DUI elements associatedwith the query terms.

According to further examples, a server to develop a searchableknowledge source for identification of distributed user interface (DUI)elements is described. An example server may include a memory and aprocessor coupled to the memory, the processor executing a knowledgesource application. The knowledge source application may be configuredto receive a control record of a source application and retrieve anapplication description associated with the source application. Theknowledge source application may also be configure to populate one ormore searchable knowledge sources with keywords, input elements, andoutput elements obtained from the application description associatedwith the source application.

According to some embodiments, a server to identify distributed userinterface (DUI) elements through a searchable knowledge source isdescribed. An example server may include a memory and a processorcoupled to the memory, the processor executing a search application. Thesearch application may be configured to receive keywords, inputelements, and output elements associated with a target workflow. Thesearch application may also be configured to execute a query on thesearchable knowledge source with query terms based on the receivedkeywords, input elements, and output elements, where the searchableknowledge source may be generated from control records of a plurality ofapplications. The search application may be further configured toprovide a query result to a requesting client, the query resultincluding one or more DUI elements associated with the query terms.

According to other embodiments, a computer-readable memory device withinstructions stored thereon is described, which when executed on one ormore computing devices may execute a method to develop a searchableknowledge source implemented to identify distributed user interface(DUI) elements. The method may be similar to the example method todevelop a searchable knowledge source implemented to identifydistributed user interface (DUI) elements discussed above.

According to further embodiments, a computer-readable memory device withinstructions stored thereon is described, which when executed on one ormore computing devices may execute a method to identify distributed userinterface (DUI) elements through implementation of a searchableknowledge source. The method may be similar to the example method toidentify distributed user interface (DU) elements through implementationof a searchable knowledge source as discussed above.

The foregoing summary is illustrative only and is not intended to be inany way limiting. In addition to the illustrative aspects, embodiments,and features described above, further aspects, embodiments, and featureswill become apparent by reference to the drawings and the followingdetailed description.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other features of this disclosure will become morefully apparent from the following description and appended claims, takenin conjunction with the accompanying drawings. Understanding that thesedrawings depict only several embodiments in accordance with thedisclosure and are, therefore, not to be considered limiting of itsscope, the disclosure will be described with additional specificity anddetail through use of the accompanying drawings, in which:

FIG. 1 illustrates an example network environment in which a searchableknowledge source may be implemented to identify distributed userinterface (DU) elements;

FIG. 2 illustrates an example DUI ecosystem;

FIG. 3 illustrates an example system to identify DUI elements;

FIG. 4 illustrates a general purpose computing device, which may be usedto develop and implement a searchable knowledge source to identify DUIelements;

FIG. 5 is a flow diagram illustrating an example method to develop andimplement a searchable knowledge source to identify DUI elements thatmay be performed by a computing device such as the computing device inFIG. 4; and

FIG. 6 illustrates a block diagram of an example computer programproduct, all arranged in accordance with at least some embodimentsdescribed herein.

DETAILED DESCRIPTION

In the following detailed description, reference is made to theaccompanying drawings, which form a part hereof. In the drawings,similar symbols typically identify similar components, unless contextdictates otherwise. The illustrative embodiments described in thedetailed description, drawings, and claims are not meant to be limiting.Other embodiments may be utilized, and other changes may be made,without departing from the spirit or scope of the subject matterpresented herein. The aspects of the present disclosure, as generallydescribed herein, and illustrated in the Figures, can be arranged,substituted, combined, separated, and designed in a wide variety ofdifferent configurations, all of which are explicitly contemplatedherein.

This disclosure is generally drawn, inter alia, to methods, apparatus,systems, devices, and/or computer program products related toidentification of distributed user interface (DUI) elements through asearchable knowledge source.

Briefly stated, technologies are generally described to develop andimplement a searchable knowledge source to identify DUI elements. Insome examples, a DUI identification system may receive a control recordof an application and populate one or more searchable knowledge sourcesbased on an application description. The application description mayinclude keywords, input elements, and output elements, and thesearchable knowledge sources may be generated from control records of amultitude of applications. The DUI identification system may execute aquery on the searchable knowledge sources based on the receivedkeywords, input elements, and output elements associated with a targetworkflow from a requesting client. A query result that includes one ormore DUI elements may be provided to the requesting client. The DUIelements may connect the input elements to corresponding output elementsand match the keywords associated with the target workflow.

A control record of an application, as described herein, may include anidentity of the application and control information associated with theapplication. The control information may include a description of howcontrols connect as producers, consumers, adopters, and developers toimplement application control serving. The control record may alsoinclude icon targets in the application (for example, a button) andcommand formatting.

An input element, as described herein, may be a first interface elementfrom a first application or the first application itself. The outputelement, as described herein, may be a final interface element producedwithin the first application or a final application following deliveryof the first interface element from the first application. The outputelement may also be the final application itself. The input element andthe output element may specify a data format including a media stream,an extensible markup language (XML) format, a uniform resource locator(URL), a keg, a graphical format, file format, text, hypertext markuplanguage (HTML), a tuple and/or one or more controls linked to a servedcontrol. Alternately, the input elements and output elements mayidentify a first application and a final application, if data formatsare not known.

A workflow, as described herein, may be a sequence of one or moreactions enabled by a control panel and/or a collection of one or morecompatible DUI elements that are or may be linked together. A workflowmay be defined by identifying an input element that may be fed into theworkflow to transpire a particular output element. For example, aworkflow may capture a photo at a first application and deliver thephoto to a second application to transpire a home video. The inputelement may be a jpeg from a camera application and the output elementmay be a media stream from a home video application.

A DUI identification system, as described herein, may include one ormore servers and one or more searchable knowledge sources, where asearchable knowledge source may be a database, a data cube, a key-valuestore, table(s), or comparable data store. The one or more servers maybe configured to execute a knowledge source application configured toreceive control records, retrieve application descriptions, and populatethe searchable knowledge sources. The servers may also be configured toexecute a search application to receive keywords, input elements, andoutput elements from a requesting client, execute a query, and provide aquery result to the client.

FIG. 1 illustrates an example network environment in which a searchableknowledge source may be implemented to identify am elements, arranged inaccordance with at least some embodiments described herein.

As shown in a diagram 100, a network 102 may host a DUI ecosystem 104and a DUI identification system 108. The DUI ecosystem 104 may includeone or more servers 106. The DUI identification system 108 may includeone or more servers 110 and one or more searchable knowledge sources112. The DUI ecosystem 104 may be coupled to a source application and aserved control as shown below in FIG. 2.

The network 102 may comprise any topology of servers, clients, Internetservice providers, and communication media, where the topology may bestatic or dynamic. The network 102 may include secure networks such asan enterprise network, an unsecure network such as a wireless opennetwork, or the Internet. The network 102 may also coordinatecommunication over other networks such as Public Switched TelephoneNetwork (PSTN) or cellular networks. Furthermore, the network 102 mayinclude short range wireless networks such as Bluetooth or similar ones.By way of example, and not limitation, the network 102 may includewireless media such as acoustic, radio frequency (RF), infrared andother wireless media.

The servers 110 of the DUI identification system 108 may receive acontrol record of the source application from the DUI ecosystem 104through a knowledge source application executed by the servers 110. Thecontrol record may include an identity and control informationassociated with the source application, where a monitor within the DUIecosystem 104 may observe the control information to provide data forthe control record.

Within the DUI identification system 108, the knowledge sourceapplication may retrieve an application description associated with thesource application using the control record. The application descriptionmay be retrieved by submitting a query to a search engine, by accessingdata on a device hosting the source application, or by performingcomparable text search approaches based on information retrieved formthe control record such as application name, file name, etc. Forexample, the application description may be retrieved by submitting aquery, with the camera application as a query term, to an existingsearch engine to retrieve data on the camera application. The data mayinclude functions of the application, how the application may interactwith other applications, descriptions of functions or uses, inputelements and output elements of the applications, and the like. Inanother example, data on a mobile device hosting a camera applicationmay be accessed to retrieve information, such as which one or more otherapplications on the mobile device have been compatible with the cameraapplication. Information may further include which output elements ofthe camera application were an input element of the other applicationsenabling compatibility. The application description may includekeywords, input elements, and output elements associated with the sourceapplication. The knowledge source application may then populate thesearchable knowledge sources 112 based on the application description,where the searchable knowledge sources 112 may include a database or adata cube or the like. The searchable knowledge sources 112 may begenerated from control records of a multitude of source applicationsenabling implementation of the searchable knowledge sources 112 toidentify DUI elements through execution of a search application.

Current web search engines and application search engines are generallykeyword based. Thus, further population of searchable knowledge sourceswith input elements and output elements along with keywords may enableeasier identification and selection of DUI elements compared to currentsearch engine techniques for non-technical users and the DUI ecosystemmay be enhanced.

In an example scenario, for a target workflow, a user may want to use agraphics system on a mobile device to apply repeatedly a single functionthat writes text along a path without having to install a largeapplication for the graphics system on the mobile device. An inputelement may be line detection in a captured photo and an output elementmay be text written along a curve in the captured photo.

In a conventional system, the user may have to capture the photo withthe mobile device, deliver the photo to a desktop computer, and am acomputer vision program that detects e shapes. One or more of thoselines may be fed into a graphics application to apply a tool that writestext along the lines. The completed photo with text may then be returnedto the mobile device for use in social network applications, emailing,and the like.

In a system according to example embodiments, the user may providekeywords, input elements, and output elements associated with the targetworkflow, such as the example elements described above, to a searchapplication executed by a server of the DUI identification system 108. Aquery engine of the DUI identification system 108 may execute a querythrough the search application based on the keywords, input elements,and output elements associated with the target workflow to provide aquery result including one or more DUI elements to the mobile device.Furthermore, compatible DUI elements may be linked together enabling themobile device to assemble a control panel to achieve the targetworkflow. In response to the user selecting one or more of the DUIelements provided in the query result to populate the mobile device, theDUI identification system 108 may provide information to implement theselected DUI elements or automatically implement the selected DUIelements on the mobile device. As a result, the user may achieve theentire target workflow on the mobile device without having to installthe large application for the graphics system. In another example thelarge applications may be installed and the user may be insulated fromthe complexity of the application by being presented with a limitedinterface element as part of their workflow.

FIG. 2 illustrates an example DUI ecosystem, arranged in accordance withat least some embodiments described herein.

As shown in a diagram 200, an example DUI ecosystem may include a sourceapplication 202, a DUI system 204, and a served control 206. A monitor208 may be located in the DUI system 204, as illustrated, the sourceapplication 202, or the served control 206. In some embodiments, themonitor 208 may not be needed as the DUI system 204 may be configured toperform the monitor's actions. The source application 202 may be adesktop application, a web page element, a web application, a packagedapplication, a ubiquitous computing (ubicomp) device service, and/or auser interface element that is isolated by the DUI system 204. The amsystem 204 may be browser based, operating system (OS) based, and/or aspecific DUI application that is activated to access one or more DUIelements.

The monitor 208 may be configured to observe control informationassociated with the source application 202 to provide data for a controlrecord of the source application 202. The control information mayinclude a description of how controls connect as producers, consumers,adopters, and developers to implement application control serving. Thecontrol record may also include an identity of the source application202, as well as icon targets in the source application 202 (for example,a button) and command formatting. The monitor 208 may provide thecontrol record of the source application 202 to a DUI identificationsystem through a knowledge source application executed by one or moreservers of the DUI identification system. The knowledge sourceapplication may retrieve an application description and populate the oneor more searchable knowledge sources based on the applicationdescription. The application description may include keywords, inputelements, and output elements associated with the source application202. In some examples, the knowledge source application may then providethe control record of the source application 202 back to the DUI system204, where the control record may include implementation information foruse of the DUI system 204. The searchable knowledge sources may then beimplemented to identify DUI elements through execution of a query.

For example, a query engine of the DUI identification system may executea query on the searchable knowledge sources through the searchapplication when a requesting client may want to assemble a controlpanel to achieve a target workflow or a goal. The client may alreadyhave one or more applications to use in the control panel and/or inputand output elements to use, but additional components may be needed toget from one application to another or from an input form to an outputform. The client may assemble the control panel for personal tasks, suchas modifying photos and sending them to social media, and/or forprofessional tasks, such as enterprise tasks.

Example embodiments may enable identification and implementation of DUIelements into the target workflow that a less technically oriented usermay otherwise not know about. Without a specific identification systemfor DUI elements, the user may have to make an uninformed pick of aprogram they know performs a particular function. The user may thenattempt to manipulate the program with DUI frameworks and tricks toenable the function to be compatible with other toots and/or functionsof the same program or a different program the user knows about.Embodiments may enable identification of existing DUI elementscompatible to functions within the target workflow by connecting inputelements to corresponding output elements, and implementation of the DUIelements in an informed and guided manner through the DUI identificationsystem.

FIG. 3 illustrates an example system to identify DUI elements, arrangedin accordance with at least some embodiments described herein.

As shown in a diagram 300, a DUI ecosystem 302 may include a sourceapplication 304, a DUI system 306, a served control 308, and othercontrols 310. A monitor 312 may be within the DUI system 306, the sourceapplication 304, the served control 308, or as part of the interfacebetween elements as illustrated. A DUI identification system 314 mayinclude one or more servers for example, a server 318) configured toexecute a knowledge source application and a search application, one ormore searchable knowledge sources 320, 322, 324, and a query engine 326.

The monitor 312 may be configured to observe control informationassociated with the source application 304 to provide data for a controlrecord 316 of the source application 304. In some embodiments, themonitor may not be present, and the DUI system 306 may be configured toobserve the control information. The control record 316 may include anidentity of the source application 304 and the control informationobserved by the monitor 312 and/or the DUI system 306. The controlinformation may include a description of how controls connect asproducers, consumers, adopters, and developers to implement applicationcontrol serving. The control record may also include icon targets in thesource application 304 (for example, a button) and command formatting.

The DUI identification system 314 may receive the control record 316from the monitor 312 and/or the DUI system 306 through the knowledgesource application executed by the server 318. The control record 316may include description of technical parameters, and keywords may comefrom a search engine by looking for the application name or file name,for example, from the control record. The search may yield keywords,input elements, and output elements associated with the sourceapplication 304. The knowledge source application may retrieve theapplication description by submitting a query to a search engine, byaccessing data on a device hosting the source application 304, or byperforming other text search techniques.

The knowledge source application may populate the searchable knowledgesources 320, 322, 324 based on the application description. For example,the knowledge source application may populate searchable knowledgesource 320 based on keywords, searchable knowledge source 322 based oninput elements, and searchable knowledge source 324 based on outputelements from the application description associated with the sourceapplication 304. In some examples, the knowledge source application maythen provide the control record of the source application 304 back tothe DUI system 306, where the control record includes implementationinformation for client use of the DUI system 306.

The DUI identification system 314 may receive keywords, input elements,and/or output elements associated with a target workflow from arequesting client through the search application executed by theservers. A query 328 may then be executed on the searchable knowledgesources 320, 322, 324 by the query engine 326, with query terms based onthe received keywords, input elements, and/or output elements associatedwith the target workflow. A query result including one or more DUIelements associated with the query terms may be provided to therequesting client through the search application. The query result mayspecifically provide DUI elements that connect the input elements withcorresponding output elements and match the keywords associated with thetarget workflow to the requesting client. The DUI elements may beassociated with web applications, open source applications, and/ordevice software. In response to the selection of one or more of the DUIelements provided from the query result, the DUI identification system314 may provide information to implement the selected DUI elements orautomatically implement the selected DUI elements within the DUIecosystem 302.

In some embodiments, the query result may provide two or more DUIelements that are compatible with one another. The compatible DUIelements may be linked together enabling assembly of a control panel toachieve the target workflow. In further embodiments, the DUI elementsprovided by the query result may be associated with applications thatare not DUI capable. In response, a secondary service and/or a programmay be applied to provide DUI capabilities to the applications that arenot DUI capable.

In other embodiments, the DUI identification system 314 may also receivethe other controls 310 within the DUI ecosystem 302 associated with thetarget workflow from the requesting client through the searchapplication executed by the servers. For example, the other controls 310may be a source application and a final application within the targetworkflow. A query 328 may be executed on the searchable knowledgesources 320, 322, 324 by the query engine 326 through the searchapplication. The query terms may be based on the other controls 310 asinput elements associated with the target workflow. The query engine 326may accept as an input element the identity of a control that the clientneeds to use in the target workflow because the output of one elementmay be the input of another element within the target workflow.Identification of workflow paths the user is not aware of may also beenabled. For example, if the client is a smart phone and a workflow froma camera application is intended to be used for an edge finding DUIelement, the client may indicate the camera application as the inputcontrol. The query engine 326 may then execute a query on the searchableknowledge sources 320, 322, 324 for various output controls of thecamera application such as a clipboard, a file system, a data exchangeservice, or a similar output control. The alternate embodiment forclients to provide other controls 310 to the DUI identification system314, may enable users with limited technical expertise to determinewhich data formats of input elements or output elements are available tosuccessfully receive a query result from the query engine 326.

In an example scenario, a user may want a photo function on their mobiledevice that is not offered by any existing application. For example, atravel agency marketing person may want to add logos and/or captions insimulated three-dimensional (3D) rendered letters that follow prominentedge paths to generate a series of travel photos with branding thatlooks like the letters have been placed physically in the environment.The user's mobile device may be able to capture the pictures, an imageprocessing program may recognize edge paths, and a graphics program mayinsert captions along arbitrary curves.

The DUI identification system 314 may receive from the user's mobiledevice keywords, such as “edge recognize,” “edge filter,” or “pathextraction,” and an input element, such as a camera source applicationthrough the search application executed by the servers. If the travelagency developer hosts the graphics program on their website, forexample, the user may also provide the graphics program as an outputelement to the DUI identification system 314. The query engine 326 maythen execute a query on the searchable knowledge sources 320, 322, 324through the search application for the keywords with the various typesof input elements and output elements specified by the user. The searchapplication may then provide a list of possible DUI elements from thequery engine 326 to the user's mobile device. Some of the am elementsmay be dependent on the mobile device having specific software executingon a desktop to export a control, while others may be available as webapplications of various sorts or standalone open source applications.The user may select one or more of the DUI elements that may enabletheir target workflow to be achieved and assemble together theappropriate components within a control panel, for example, with thefinal output going to the mobile device “share to” intent.

Accordingly, anytime the user takes a photo, they may be enabled to grabthe photo and find edges. They may then erase some of the edges on themobile device and/or add their branding caption text into the photoalong the resulting edges. The modified photo may then be shared with athird party social networking site. Assembling such a workflow mayenable a client to quickly perform a series of steps on their mobiledevice that may normally take multiple discrete programs and specializedskills at a desktop. An arbitrary number of workflows may be assembledinto small widget groups of associated DUI elements once such functionscan be identified.

The examples in FIGS. 1 through 3 have been described using specific DUIsystems and processes to identify DUI elements through development andimplementation of a searchable knowledge source. Embodiments to identifyDUI elements are not limited to the specific DUI systems and processesaccording to these examples.

FIG. 4 illustrates a general purpose computing device, which may be usedto which may be used to develop and implement a searchable knowledgesource to identify DUI elements, arranged in accordance with at leastsome embodiments described herein.

For example, the computing device 400 may be used to implement asearchable knowledge source to identify DUI elements, as describedherein. In an example basic configuration 402, the computing device 400may include one or more processors 404 and a system memory 406. A memorybus 408 may be used for communicating between the processor 404 and thesystem memory 406. The basic configuration 402 is illustrated in FIG. 4by those components within the inner dashed line.

Depending on the desired configuration, the processor 404 may be of anytype, including but not limited to a microprocessor (μP), amicrocontroller (μC), a digital signal processor (DSP), or anycombination thereof. The processor 404 may include one more levels ofcaching, such as a level cache memory 412, one or more processor cores414, and registers 416. The example processor cores 414 may (each)include an arithmetic logic unit (ALU), a floating point unit (FPU), adigital signal processing core (DSP Core), or any combination thereof.An example memory controller 418 may also be used with the processor404, or in some implementations, the memory controller 418 may be aninternal part of the processor 404.

Depending on the desired configuration, the system memory 406 may be ofany type including but not limited to volatile memory (such as RAM),non-volatile memory (such as ROM, flash memory, etc.) or any combinationthereof. The system memory 406 may include an operating system 420, aDUI identification application 422, and program data 424. The DUIidentification application 422 may include a population module 426 and aquery module 427. The population module 426 may populate one or moresearchable knowledge sources based on an application descriptionassociated with a source application. The application description may beretrieved from a control record, the application description includingkeyword, input elements, and output elements. The query module 427 mayexecute a query on the searchable knowledge sources with query termsbased on keywords, input elements, and output elements received by a DUIidentification system from a requesting client. The program data 424 mayinclude, among other data, query data 428 related to a query resultproduced by the query module 427, as described herein.

The computing device 400 may have additional features or functionality,and additional interfaces to facilitate communications between the basicconfiguration 402 and any desired devices and interfaces. For example, abus/interface controller 430 may be used to facilitate communicationsbetween the basic configuration 402 and one or more data storage devices432 via a storage interface bus 434. The data storage devices 432 may beone or more removable storage devices 436, one or more non-removablestorage devices 438, or a combination thereof. Examples of the removablestorage and the non-removable storage devices include magnetic diskdevices such as flexible disk drives and hard-disk drives (HDD), opticaldisk drives such as compact disk (CD) drives or digital versatile disk(DVD) drives, solid state drives (SSD), and tape drives to name a few.Example computer storage media may include volatile and nonvolatile,removable and non-removable media implemented in any method ortechnology for storage of information, such as computer readableinstructions, data structures, program modules, or other data.

The system memory 406, the removable storage devices 436 and thenon-removable storage devices 438 are examples of computer storagemedia. Computer storage media includes, but is not limited to, RAM, ROM,EEPROM, flash memory or other memory technology, CD-ROM, digitalversatile disks (DVD), solid state drives, or other optical storage,magnetic cassettes, magnetic tape, magnetic disk storage or othermagnetic storage devices, or any other medium which may be used to storethe desired information and which may be accessed by the computingdevice 400. Any such computer storage media may be part of the computingdevice 400.

The computing device 400 may also include an interface bus 440 forfacilitating communication from various interface devices (for exampleone or more output devices 442, one or more peripheral interfaces 444,and one or more communication devices 466) to the basic configuration402 via the bus/interface controller 430. Some of the example outputdevices 442 include a graphics processing unit 448 and an audioprocessing unit 450, which may be configured to communicate to variousexternal devices such as a display or speakers via one or more A/V ports452. One or more example peripheral interfaces 444 may include a serialinterface controller 454 or a parallel interface controller 456, whichmay be configured to communicate with external devices such as inputdevices (for example, keyboard, mouse, pen, voice input device, touchinput device, etc.) or other peripheral devices (for example, printer,scanner, etc.) via one or more I/O ports 458. An example communicationdevice 466 includes a network controller 460, which may be arranged tofacilitate communications with one or more other computing devices 462over a network communication link via one or more communication ports464. The one or more other computing devices 462 may include servers,client devices, and comparable devices.

The network communication link may be one example of a communicationmedia. Communication media may typically be embodied by computerreadable instructions, data structures, program modules, or other datain a modulated data signal, such as a carrier wave or other transportmechanism, and may include any information delivery media. A “modulateddata signal” may be a signal that has one or more of its characteristicsset or changed in such a manner as to encode information in the signal.By way of example, and not limitation, communication media may includewired media such as a wired network or direct-wired connection, andwireless media such as acoustic, radio frequency (RF), microwave,infrared (IR) and other wireless media. The term computer readable mediaas used herein may include both storage media and communication media.

The computing device 400 may be implemented as a part of a generalpurpose or specialized server, mainframe, or similar computer thatincludes any of the above functions. The computing device 400 may alsobe implemented as a personal computer including both laptop computer andnon-laptop computer configurations.

Example embodiments may also include methods to develop a searchableknowledge source to identify DUI elements. These methods can beimplemented in any number of ways, including the structures describedherein. One such way may be by machine operations, of devices of thetype described in the present disclosure. Another optional way may befor one or more of the individual operations of the methods to beperformed in conjunction with one or more human operators performingsome of the operations while other operations may be performed bymachines. These human operators need not be collocated with each other,but each can be only with a machine that performs a portion of theprogram. In other embodiments, the human interaction can be automatedsuch as by pre-selected criteria that may be machine automated.

FIG. 5 is a flow diagram illustrating an example method to develop andimplement a searchable knowledge source to identify DUI elements thatmay be performed by a computing device such as the computing device inFIG. 4, arranged in accordance with at least some embodiments describedherein.

Example methods may include one or more operations, functions or actionsas illustrated by one or more of blocks 522, 524, 526, 528, 530, and532. The operations described in the blocks 522 through 532 may also bestored as computer-executable instructions in a computer-readable mediumsuch as a computer-readable medium 520 of a computing device 510.

An example process to develop and implement a searchable knowledgesource to identify DUI elements may begin with block 522, “RECEIVE ACONTROL RECORD OF A SOURCE APPLICATION,” where a DUI identificationsystem (for example, the DUI identification system 314) may receive acontrol record (for example, the control record 316) of a sourceapplication (for example, the source application 304) from a DUIecosystem (for example, the DUI ecosystem 302) through a knowledgesource application executed by one or more servers of the DUIidentification system. A monitor (for example, the monitor 312) withinthe am ecosystem may be configured to observe control informationassociated with the source application to provide data for the controlrecord. The monitor may be located in a DUI system (for example, the DUIsystem 306), the source application, or a served control (for example,the served control 308).

Block 522 may be followed by block 524, “RETRIEVE AN APPLICATIONDESCRIPTION ASSOCIATED WITH THE SOURCE APPLICATION,” where the knowledgesource application may retrieve an application description associatedwith the source application. The application description may beretrieved by submitting a query to a search engine, accessing files on adevice hosting the source application, and/or performing other textsearch techniques. The application description may include keywords,input elements, and output elements associated with the sourceapplication.

Block 524 may be followed by block 526, “POPULATE ONE OR MORE SEARCHABLEKNOWLEDGE SOURCES BASED ON THE APPLICATION DESCRIPTION,” where theknowledge source application may populate one or more searchableknowledge sources (for example, the searchable knowledge sources 320,322, 324) based on the application description. The searchable knowledgesources may be further developed from control records of a multitude ofsource applications.

Block 526 may be followed by block 528, “RECEIVE KEYWORDS, INPUTELEMENTS, AND OUTPUT ELEMENTS ASSOCIATED WITH A TARGET WORKFLOW,” wherethe servers of the DUI identification system may execute a searchapplication to receive keywords, input elements, and output elementsassociated with a target workflow from a requesting client.

Block 528 may be followed by block 530, “EXECUTE A QUERY ON THESEARCHABLE KNOWLEDGE SOURCES WITH QUERY TERMS BASED ON THE RECEIVEDKEYWORDS, INPUT ELEMENTS, AND OUTPUT ELEMENTS,” where the query engine(for example, the query engine 326) of the DUI identification system mayexecute a query (for example, the query 328) on the searchable knowledgesources through the search application. Query terms may be based on thereceived keywords, input elements, and output elements associated withthe target workflow.

Block 530 may be followed by block 532, “PROVIDE A QUERY RESULT TO AREQUESTING CLIENT, THE QUERY RESULT INCLUDING ONE OR MORE DUI ELEMENTSASSOCIATED WITH THE QUERY TERMS,” where the search application mayprovide a query result to the requesting client. The query result mayinclude one or more DUI elements associated with the query terms, wherethe DUI elements may connect the input elements to corresponding outputelements and match keywords associated with the target workflow. Inother embodiments, the query result may provide two or more DUI elementsthat are compatible with one another and may be linked together toenable the client to assemble a control panel to achieve the targetworkflow.

The blocks included in the above described process are for illustrationpurposes. Identification of DUI elements through development andimplementation of searchable knowledge sources may be implemented bysimilar processes with fewer or additional blocks. In some embodiments,the blocks may be performed in a different order. In some otherembodiments, various blocks may be eliminated. In still otherembodiments, various blocks may be divided into additional blocks, orcombined together into fewer blocks.

FIG. 6 illustrates a block diagram of an example computer programproduct, arranged in accordance with at least some embodiments describedherein.

In some embodiments, as shown in FIG. 6, the computer program product600 may include a signal bearing medium 602 that may also include one ormore machine readable instructions 604 that, when executed by, forexample, a processor, may provide the functionality described herein.Thus, for example, referring to the processor 404 in FIG. 4, apopulation module 426 and a query module 427 executed on the processor404 may undertake one or more of the tasks shown in FIG. 6 in responseto the instructions 604 conveyed to the processor 404 by the medium 602to perform actions associated with establishing secure communications tomanage components of a control system as described herein. Some of thoseinstructions may include, for example, instructions for receiving acontrol record of a source application, retrieving an applicationdescription associated with the source application, populating one ormore searchable knowledge sources based on the application description,receiving keywords, input elements, and output elements associated witha target workflow, executing a query on the searchable knowledge sourcewith query terms based on the keywords, the input elements, and theoutput elements, and providing a query result to a requesting client,the query result including one or more DUI elements associated with thequery terms, according to some embodiments described herein.

In some implementations, the signal bearing medium 602 depicted in FIG.6 may encompass a computer-readable medium 606, such as, but not limitedto, a hard disk drive, a solid state drive, a Compact Disc (CD), aDigital Versatile Disk (DVD), a digital tape, memory, etc. In someimplementations, the signal bearing medium 602 may encompass arecordable medium 608, such as, but not limited to, memory, read/write(R/W) CDs, R/W DVDs, etc. In some implementations, the signal bearingmedium 602 may encompass a communications medium 610, such as, hut notlimited to, a digital and/or an analog communication medium (forexample, a fiber optic cable, a waveguide, a wired communications link,a wireless communication link, etc.). Thus, for example, the programproduct 600 may be conveyed to one or more modules of the processor 404of FIG. 4 by an RF signal bearing medium, where the signal bearingmedium 602 is conveyed by the wireless communications medium 610 (forexample a wireless communications medium conforming with the IEEE 802.11standard).

According to some examples, a method to develop a searchable knowledgesource for identification of distributed user interface (DUI) elementsis provided. An example method may include receiving a control record ofa source application and retrieving an application descriptionassociated with the source application. The method may further includepopulating one or more searchable knowledge sources based on theapplication description.

In other examples, the control record of the source application mayinclude an identity of the source application and control informationassociated with the source application. The control record of the sourceapplication may be received from a monitor, where the monitor may belocated in one of a DUI system, the source application, and a servedcontrol. The monitor may be configured to observe control informationassociated with the source application to provide data for the controlrecord. A query may be submitted to a search engine to retrieve theapplication description. Data may be accessed on a device hosting thesource application to retrieve the application description.

In further examples, the one or more searchable knowledge sources may bepopulated with keywords, input elements, and output elements obtainedfrom the application description associated with the source application,where the searchable knowledge sources are a database or a data cube.The control record of the source application may be provided to a DUIsystem, the control record including implementation information forclient use of the DUI system.

According to some embodiments, a method to identify distributed userinterface (DUI) elements through a searchable knowledge source isprovided. An example method may include receiving keywords, inputelements, and output elements associated with a target workflow. Themethod may also include executing a query on the searchable knowledgesource with query terms based on the received keywords, input elements,and output elements, where the searchable knowledge source may begenerated from control records of a plurality of applications. Themethod may further include providing a query result to a requestingclient, the query result including one or more DUI elements associatedwith the query terms.

In other embodiments, the query result may further include one or moreDUI elements that connect one or more input elements to correspondingone or more output elements and match one or more keywords associatedwith the target workflow. In response to a selection of one or more DUIelements from the query result to implement, information to implementthe one or more selected DUI elements is provided or the one or moreselected DUI elements are automatically implemented within a DUI system.

In further embodiments, a query result may provide two or more DUIelements that are compatible with one another. The two or morecompatible DUI elements may be linked together enabling assembly of acontrol panel to achieve the target workflow from the compatible DUIelements. The searchable knowledge source may be a database or a datacube.

According to some examples, a server to develop a searchable knowledgesource for identification of distributed user interface (DUI) elementsis described. An example server may include a memory and a processorcoupled to the memory, the processor executing a knowledge sourceapplication. The knowledge source application may be configured toreceive a control record of a source application and retrieve anapplication description associated with the source application. Theknowledge source application may also be configured to populate one ormore searchable knowledge sources with keywords, input elements, andoutput elements obtained from the application description associatedwith the source application.

In other examples, the source application may include a desktopapplication, a web page element, a web application, a ubiquitouscomputing (ubicomp) device service, or a user interface element that isisolated by a DUI system. The DUI system may be browser based, operatingsystem (OS) based, and/or a specific DUI application that is activatedto access one or more DUI elements. The control record of the sourceapplication may include an identity of the source application andcontrol information associated with the source application, where thecontrol information associated with the source application includes adescription of how controls connect as producers, consumers, adopters,and developers to implement application control serving. The controlrecord of the source application may further include icon targets in thesource application and command formatting. The control record of thesource application may be received from a monitor, where the monitor maybe located in a DUI system, the source application, or a served control.The monitor may be configured to observe control information associatedwith the source application to provide data for the control record.

In further examples, the knowledge source application may be configuredto retrieve the application description through a search engine and/ordata accessed on a device hosting the source application. The inputelements may be first interface elements from a first source applicationthat are delivered to second interface elements to produce the outputelements on one of the first source application and a second sourceapplication. The input elements and the output elements may specify adata format, where the data format may include a media stream, an XML, aURL, a jpeg, a graphical format, text, HTML, tuple and/or one or morecontrols linked to a served control. The knowledge source applicationmay be further configured to provide the control record of the sourceapplication to a DUI system, the control record including implementationinformation for use of the DUI system.

According to some embodiments, a server to identify distributed userinterface (DUI) elements through a searchable knowledge source isdescribed. An example server may include a memory and a processorcoupled to the memory, the processor executing a search application. Thesearch application may be configured to receive keywords, inputelements, and output elements associated with a target workflow. Thesearch application may also be configured to execute a query on thesearchable knowledge source with query terms based on the receivedkeywords, input elements, and output elements, where the searchableknowledge source may be generated from control records of a plurality ofapplications. The search application may be further configured toprovide a query result to a requesting client, the query resultincluding one or more DUI elements associated with the query terms.

In other embodiments, the query result may further include one or moreDUI elements that connect one or more input elements to correspondingone or more output elements and match one or more keywords associatedwith the target workflow. The input elements may be first interfaceelements from a first source application that are delivered to secondinterface elements to produce the output elements on one of the firstsource application and a second source application. The input elementsand the output elements may specify a data format, where the data formatmay include a media stream, an XML, a URL, a jpeg, a graphical format,text, HTML, a tuple and/or one or more controls linked to a servedcontrol. In response to a selection of one or more DUI elements from thequery result to implement, the search application may be configured toprovide information to the client to implement the one or more selectedDUI elements or the search application may be configured to enableautomatic implementation of the one or more selected DUI elements withina DUI system.

In further embodiments, the query result may further provide two or moreDUI elements that are compatible with one another, where the searchapplication may be configured to link the two or more compatible DUIelements together. The search application may also be configured toenable assembly of a control panel to achieve the target workflow fromthe two or more compatible DUI elements. The DUI elements provided bythe query result may be associated with web applications, open sourceapplications, or device software. The DUI elements provided by the queryresult may also be associated with applications that are not DUIcapable, where a secondary service or a program may be applied toprovide DUI capabilities to the applications that are not DUI capable.

According to other examples, a computer-readable memory device withinstructions stored thereon is described, which when executed on one ormore computing devices may execute a method to develop a searchableknowledge source implemented to identify distributed user interface(DUI) elements. The method may be similar to the example method todevelop a searchable knowledge source implemented to identifydistributed user interface (DUI) elements discussed above.

According to other embodiments, a computer-readable memory device withinstructions stored thereon is described, which when executed on one ormore computing devices may execute a method to identify distributed userinterface (DUI) elements through implementation of a searchableknowledge source. The method may be similar to the example method toidentify distributed user interface (DUI) elements throughimplementation of a searchable knowledge source as discussed above.

There is little distinction left between hardware and softwareimplementations of aspects of systems; the use of hardware or softwareis generally (but not always, in that in certain contexts the choicebetween hardware and software may become significant) a design choicerepresenting cost vs. efficiency tradeoffs. There are various vehiclesby which processes and/or systems and/or other technologies describedherein may be effected (for example, hardware, software, and/orfirmware), and that the preferred vehicle will vary with the context inwhich the processes and/or systems and/or other technologies aredeployed. For example, if an implementer determines that speed andaccuracy are paramount, the implementer may opt for a mainly hardwareand/or firmware vehicle; if flexibility is paramount, the implementermay opt for a mainly software implementation; or, yet againalternatively, the implementer may opt for some combination of hardware,software, and/or firmware.

The foregoing detailed description has set forth various examples of thedevices and/or processes via the use of block diagrams, flowcharts,and/or examples. Insofar as such block diagrams, flowcharts, and/orexamples contain one or more functions and/or operations, it will beunderstood by those within the art that each function and/or operationwithin such block diagrams, flowcharts, or examples may be implemented,individually and/or collectively, by a wide range of hardware, software,firmware, or virtually any combination thereof. In one embodiment,several portions of the subject matter described herein may beimplemented via Application Specific Integrated Circuits (ASICs), FieldProgrammable Gate Arrays (FPGAs), digital signal processors (DSPs), orother integrated formats. However, those skilled in the art willrecognize that some aspects of the embodiments disclosed herein, inwhole or in part, may be equivalently implemented in integratedcircuits, as one or more computer programs running on one or morecomputers (for example as one or more programs running on one or morecomputer systems), as one or more programs running on one or moreprocessors (for example, as one or more programs running on one or moremicroprocessors), as firmware, or as virtually any combination thereof,and that designing the circuitry and/or writing the code for thesoftware and or firmware would be well within the skill of one of skillin the art in light of this disclosure.

The present disclosure is not to be limited in terms of the particularexamples described in this application, which are intended asillustrations of various aspects. Many modifications and variations canbe made without departing from its spirit and scope, as will be apparentto those skilled in the art. Functionally equivalent methods andapparatuses within the scope of the disclosure, in addition to thoseenumerated herein, will be apparent to those skilled in the art from theforegoing descriptions. Such modifications and variations are intendedto fall within the scope of the appended claims. The present disclosureis to be limited only by the terms of the appended claims, along withthe full scope of equivalents to which such claims are entitled. It isalso to be understood that the terminology used herein is for thepurpose of describing particular embodiments only, and is not intendedto be limiting.

In addition, those skilled in the art will appreciate that themechanisms of the subject matter described herein are capable of beingdistributed as a program product in a variety of forms, and that anillustrative embodiment of the subject matter described herein appliesregardless of the particular type of signal bearing medium used toactually carry out the distribution. Examples of a signal bearing mediuminclude, but are not limited to, the following: a recordable type mediumsuch as a floppy disk, a hard disk drive, a Compact Disc (CD), a DigitalVersatile Disk (DVD), a digital tape, a computer memory, a solid statedrive, etc.; and a transmission type medium such as a digital and/oranalog communication medium (for example a fiber optic cable, awaveguide, a wired communications link, a wireless communication link,etc.).

Those skilled in the art will recognize that it is common within the artto describe devices and/or processes in the fashion set forth herein,and thereafter use engineering practices to integrate such describeddevices and/or processes into data processing systems. That is, at leasta portion of the devices and/or processes described herein may beintegrated into a data processing system via a reasonable amount ofexperimentation. Those having skill in the art will recognize that atypical data processing system generally includes one or more of asystem unit housing, a video display device, a memory such as volatileand non-volatile memory, processors such as microprocessors and digitalsignal processors, computational entities such as operating systems,drivers, graphical user interfaces, and applications programs, one ormore interaction devices, such as a touch pad or screen, and/or controlsystems including feedback loops and control motors.

A typical data processing system may be implemented utilizing anysuitable commercially available components, such as those typicallyfound in systems employing searchable knowledge sources to identify DUIelements. The herein described subject matter sometimes illustratesdifferent components contained within, or connected with, differentother components. It is to be understood that such depictedarchitectures are merely exemplary, and that in fact many otherarchitectures may be implemented which achieve the same functionality.In a conceptual sense, any arrangement of components to achieve the samefunctionality is effectively “associated” such that the desiredfunctionality is achieved. Hence, any two components herein combined toachieve a particular functionality may be seen as “associated with” eachother such that the desired functionality is achieved, irrespective ofarchitectures or intermediate components. Likewise, any two componentsso associated may also be viewed as being “operably connected,” or“operably coupled,” to each other to achieve the desired functionality,and any two components capable of being so associated may also be viewedas being “operably couplable,” to each other to achieve the desiredfunctionality. Specific examples of operably couplable include but arenot limited to physically connectable and/or physically interactingcomponents and/or wirelessly interactable and/or wirelessly interactingcomponents and/or logically interacting and/or logically interactablecomponents.

With respect to the use of substantially any plural and/or singularterms herein, those having skill in the art can translate from theplural to the singular and/or from the singular to the plural as isappropriate to the context and/or application. The varioussingular/plural permutations may be expressly set forth herein for sakeof clarity.

It will be understood by those within the art that, in general, termsused herein, and especially in the appended claims (for example bodiesof the appended claims) are generally intended as “open” terms (forexample, the term “including” should be interpreted as “including butnot limited to,” the term “having” should be interpreted as “having atleast,” the term “includes” should be interpreted as “includes but isnot limited to,” etc.). It will be further understood by those withinthe art that if a specific number of an introduced claim recitation isintended, such an intent will be explicitly recited in the claim, and inthe absence of such recitation no such intent is present. For example,as an aid to understanding, the following appended claims may containusage of the introductory phrases “at least one” and “one or more” tointroduce claim recitations. However, the use of such phrases should notbe construed to imply that the introduction of a claim recitation by theindefinite articles “a” or “an” limits any particular claim containingsuch introduced claim recitation to examples containing only one suchrecitation, even when the same claim includes the introductory phrases“one or more” or “at least one” and indefinite articles such as “a” or“an” (for example, “a” and/or “an” should be interpreted to mean “atleast one” or “one or more”); the same holds true for the use ofdefinite articles used to introduce claim recitations. In addition, evenif a specific number of an introduced claim recitation is explicitlyrecited, those skilled in the art will recognize that such recitationshould be interpreted to mean at least the recited number (for examplethe bare recitation of “two recitations,” without other modifiers, meansat least two recitations, or two or more recitations).

Furthermore, in those instances where a convention analogous to “atleast one of A, B, and C, etc.” is used, in general such a constructionis intended in the sense one having skill in the art would understandthe convention (for example, “a system having at least one of A, B, andC” would include but not be limited to systems that have A alone, Balone, C alone, A and B together, A and C together, B and C together,and/or A, B, and C together, etc.). It will be further understood bythose within the art that virtually any disjunctive word and/or phrasepresenting two or more alternative terms, whether in the description,claims, or drawings, should be understood to contemplate thepossibilities of including one of the terms, either of the terms, orboth terms. For example, the phrase “A or B” will be understood toinclude the possibilities of “A” or “B” or “A and B.”

As will be understood by one skilled in the art, for any and allpurposes, such as in terms of providing a written description, allranges disclosed herein also encompass any and all possible subrangesand combinations of subranges thereof. Any listed range can be easilyrecognized as sufficiently describing and enabling the same range beingbroken down into at least equal halves, thirds, quarters, fifths,tenths, etc. As a non-limiting example, each range discussed herein canbe readily broken down into a lower third, middle third and upper third,etc. As will also be understood by one skilled in the art all languagesuch as “up to,” “at least,” “greater than,” “less than,” and the likeinclude the number recited and refer to ranges which can be subsequentlybroken down into subranges as discussed above. Finally, as will beunderstood by one skilled in the art, a range includes each individualmember. Thus, for example, a group having 1-3 cells refers to groupshaving 1, 2, or 3 cells. Similarly, a group having 1-5 cells refers togroups having 1, 2, 3, 4, or 5 cells, and so forth.

While various aspects and embodiments have been disclosed herein, otheraspects and embodiments will be apparent to those skilled in the art.The various aspects and embodiments disclosed herein are for purposes ofillustration and are not intended to be limiting, with the true scopeand spirit being indicated by the following claims.

1. A method to develop a searchable knowledge source for identificationof distributed user interface (DUI) elements, the method comprising:receiving a control record of a source application from a monitor,wherein the monitor is located in one of a DUI system, the sourceapplication, and a served control; retrieving an application descriptionassociated with the source application; and populating one or moresearchable knowledge sources based on the application description. 2.(canceled)
 3. (canceled)
 4. (canceled)
 5. The method of claim 1, whereinretrieving an application description associated with the sourceapplication comprises: submitting a query to a search engine to retrievethe application description.
 6. The method of claim 1, whereinretrieving an application description associated with the sourceapplication further comprises: accessing data on a device hosting thesource application to retrieve the application description.
 7. Themethod of claim 1, wherein populating the one or more searchableknowledge sources comprises: populating the one or more searchableknowledge sources with keywords, input elements, and output elementsobtained from the application description associated with the sourceapplication.
 8. (canceled)
 9. The method of claim 1, further comprising:providing the control record of the source application to a DUI system,the control record comprising implementation information for client useof the DUI system.
 10. (canceled)
 11. (canceled)
 12. (canceled) 13.(canceled)
 14. (canceled)
 15. (canceled)
 16. (canceled)
 17. A server todevelop a searchable knowledge source for identification of distributeduser interface (DUI) elements, the server comprising: a memory; aprocessor coupled to the memory, the processor executing a knowledgesource application, the knowledge source application configured to:receive a control record of a source application, wherein the controlrecord of the source application comprises an identity of the sourceapplication and control information associated with the sourceapplication; retrieve an application description associated with thesource application; and populate one or more searchable knowledgesources with keywords, input elements, and output elements obtained fromthe application description associated with the source application. 18.(canceled)
 19. (canceled)
 20. (canceled)
 21. (canceled)
 22. The serverof claim 17, wherein the control record of the source applicationfurther comprises icon targets in the source application and commandformatting.
 23. (canceled)
 24. (canceled)
 25. The server of claim 17,wherein the control record of the source application is received from amonitor, and wherein the monitor is configured to observe controlinformation associated with the source application to provide data forthe control record.
 26. The server of claim 17, wherein the knowledgesource application is configured to retrieve the application descriptionthrough one or more of: a search engine and data accessed on a devicehosting the source application.
 27. The server of claim 17, wherein theinput elements are first interface elements from a first sourceapplication that are delivered to second interface elements to producethe output elements on one of the first source application and a secondsource application.
 28. (canceled)
 29. (canceled)
 30. The server ofclaim 17, wherein the knowledge source application is further configuredto provide the control record of the source application to a DUI system,the control record comprising implementation information for use of theDUI system.
 31. A server to identify distributed user interface (DUI)elements through a searchable knowledge source, the server comprising: amemory; a processor coupled to the memory, the processor executing asearch application, the search application configured to: receivekeywords, input elements, and output elements associated with a targetworkflow; execute a query on the searchable knowledge source with queryterms based on the received keywords, input elements, and outputelements, wherein the searchable knowledge source is generated fromcontrol records of a plurality of applications, and wherein the controlrecords are received from a monitor configured to observe controlinformation associated with the search application; and provide a queryresult to a requesting client, the query result comprising one or moreDUI elements associated with the query terms.
 32. The server of claim31, wherein the query result further comprises one or more DUI elementsthat connect one or more input elements to corresponding one or moreoutput elements and match one or more keywords associated with thetarget workflow.
 33. The server of claim 31, wherein the input elementsare first interface elements from a first source application that aredelivered to second interface elements to produce the output elements onone of the first source application and a second source application. 34.(canceled)
 35. (canceled)
 36. The server of claim 31, wherein inresponse to a selection of one or more DUI elements from the queryresult to implement, the search application is configured to provideinformation to the client to implement the one or more selected DUIelements.
 37. The server of claim 31, wherein in response to a selectionof one or more DUI elements from the query result to implement, thesearch application is configured to enable automatic implementation ofthe one or more selected DUI elements within a DUI system.
 38. Theserver of claim 31, wherein the query result further provides two ormore DUI elements that are compatible with one another.
 39. (canceled)40. The server of claim 38, wherein the search application is configuredto enable assembly of a control panel to achieve the target workflowfrom the two or more compatible DUI elements.
 41. (canceled)
 42. Theserver of claim 31, wherein the one or more DUI elements provided by thequery result are associated with applications that are not DUI capable.43. The server of claim 42, wherein one of a secondary service and aprogram is applied to provide DUI capabilities to the applications thatare not DUI capable.
 44. (canceled)
 45. (canceled)